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1 Introduction 

The main goal of the paper is to generahze the Duistermaat-Guillemin trace formula to the 
case of transversally elliptic operators on a compact foliated manifold. First, let us recall 
briefly the setting of the classical formula. 

Let P be a positive self-adjoint elliptic pseudodifferential operator of order one on a 
closed manifold M (for example, P = -\/A, where A is the Laplace-Beltrami operator of a 
Riemannian metric on M). For any function / G C^(]R), the operator Uf = j f{t)e^^^dt 
can be shown to be of trace class, and the mapping 9 : f ^ iiU f is a, continuous linear 
functional on C^(M). Otherwise speaking, ^ is a distribution on R. The principal symbol 
p of the operator P is a smooth function on the symplectic manifold T*M \ 0, and, by 
definition, the bicharacteristic flow ft of the operator P is the Hamiltonian flow on T*M \ 
defined by the function p. 

By the theorem due to Colin de Verdiere and Chazarain the singularities of the 

distribution 9 are contained in the period set of closed trajectories of the bicharacteristic 
flow /(. Moreover, Duistermaat and Guillemin showed that, under the assumption that 
the bicharacteristic flow is clean, one can write down an asymptotic expansion for the dis- 
tribution 9 near a given period of closed bicharacteristic. A formula for the leading term of 
this asymptotic expansion is the Duistermaat-Guillemin trace formula mentioned above. It 
involves the geometry of the bicharacteristic flow in the form of Poincare map and Maslov 
indices and provides a far-reaching generalization of the classical Poisson formula and the 
Selberg trace formula on hyperbolic spaces. 

In the paper, we prove a trace formula for an operator P = \fA, where A is a positive 
self-adjoint transversally elliptic pseudodifferential operator of order two with the positive, 
holonomy invariant transverse principal symbol on a compact foliated manifold (M, T) (see 
Theorem One can consider such an operator as an elliptic operator on the singular 
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space MjT of leaves of the foliation T (this statement can be made more precise, using 
the language of noncommutative geometry, see [Q) the trace formula stated in this paper 
as an example of a trace formula for elliptic operators on singular spaces. We hope that 
this formula will be useful in further study of a general trace formula in noncommutative 
geometry (see, for instance, [|], Q for discussion of a noncommutative trace formula). 

It should be also noted that our trace formula can be viewed as a relative version of the 
Duistermaat-Guillemin trace formula. 

2 Preliminaries and main results 

Let (M, JF) be a compact, connected, oriented foliated manifold. We will use the following 
notation: TT is the tangent bundle, HT = TM/TJ^ is the normal bundle and N*^ is the 
conormal bundle to JF. There is a short exact sequence 

— ^ TJ^ — ^ TM — > HJ^ — > 0. (2.1) 

We will consider linear operators, acting on half-densities. Recall that an a-density 
{a G M) on a real vector space V of dimension n is a map : A"V" M such that 0(Af ) = 
|A|°0(f),f G A"y, A G M. For any real vector bundle E over a smooth manifold X, we will 
denote by \E\^ the a-density bundle of E. 

Given a pseudodifferential operator A G \E'™(M, [TM]^^"^)^ the transversal principal 
symbol a a of A is defined to be the restriction of its principal symbol on N*J^ = N*J-'\0. 
An operator A G \1'''"(M, |TM|^/^) is said to be transversally elliptic, if cr^(z/) ^ for any 
z/ G N*T. 

For any smooth leafwise path 7 from x E M toy & M, sliding along leaves of the foliation 
defines the holonomy map h^, which associates to every germ of a local transversal to the 
foliation at the point x a germ of a local transversal to the foliation at the point y (this map 
is a natural generalization of the Poincare first-return map for flows). The differential of this 
map (the linear holonomy map) is well-defined as a linear map dh^ : Hr^J^ — > HyJ^ and the 
CO differential linear map dh^ : N*J^ 

The transversal principal symbol a a of an operator A G \E'™'(M, |TM|^/^) is said to be 
holonomy invariant, if cr^((i/i*(i/)) = ^^(z^) for any smooth leafwise path 7 from x to y 
and for any u G N*J-'. 

Throughout in the paper, we will assume that A is a linear operator in C°°(M, |TM|^/^), 
satisfying the following conditions: 

(Al) A G \Ef^(M, |TM|^/^) is a transversally elliptic operator with the positive, holonomy 
invariant transversal principal symbol; 

(A2) A is an essentially self-adjoint positive operator in space of half-densities on M, 
L2(M) (with the initial domain C°°(M, \TM\^/'^)). 

Example 1. A geometrical example of an operator, satisfying the conditions (Al) and (A2), 
is given by the operator A = I + A^, where Ah is the transversal Laplacian of a bundle-like 
metric on a Riemannian foliation. 
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Recall that a foliation on a smooth Riemannian manifold {M^qm) is Riemannian if 
it satisfies one of the following equivalent conditions (see, for instance, 0): 

1. (M, JF) locally has the structure of Riemannian submersion; 

2. the transverse part of the Riemannian metric qm (that is, its restriction to H = TJ-'^) 
is holonomy invariant; 

3. the horizontal distribution H is totally geodesic. 

In this case, the metric qm is called bundle-like. 

The Riemannian metric qm defines a decomposition of the cotangent bundle T*M into a 
direct sum T*M = F* (B H*. With respect to this decomposition, the de Rham differential 
d : C°°(M) C°°{M,T*M) can be written as a sum d = dp + dn, where dp : C~(M) 
C^{M,F*) and dn : C~(M) C^{M,H*). 

The transversal Laplacian is a second order transversally elliptic differential operator in 
the space C°°{M) defined by the formula 

A/f = —d*fjdH- 

Its principal symbol a2 is given by the formula 

and the holonomy invariance of the transverse principal symbol a^jj is equivalent to the 
assumption on the Riemannian metric qm to be bundle-like. 

From now on, we will assume that A satisfies the assumptions (Al) and (A2). By 
the spectral theorem, the operator P = a/A generates a strongly continuous group e**^ of 
bounded operators in LF'{M). To define a distributional trace of the operator e**^, one need 
an additional regularization. First, let us introduce some notation. 

Recall that the holonomy groupoid G = Gjc of the foliation JF is the set of equivalence 
classes of leafwise paths 7 : [0, 1] — M with respect to an equivalence relation setting 
7i ~h 72 if 7i and 72 have the same initial and final points and the same holonomy maps. 
G is equipped with maps s,r : G ^ M given by 5(7) = 7(0) and r(7) = 7(1) and has a 
composition law given by the composition of paths. For any 71,72 € G, the composition 
7i o 72 makes sense iff r(72) = ^(71). We will make use of standard notation: G^ = r~^{x), 
Gx = s~^{x), G^ = s'^i^x) n r~^{x), x G M. For any x G M, G^ is the holonomy group of 
the leaf through the point x and the maps s : G^' and r : Gx — >■ are covering 

maps associated with G^. We will identify a point x G M with the element in G given by 
the constant path ■y(t) = x, t G [0, 1]. 

Let s*(|TjF|^/^) and r*(|TjF|^/^) be the lifts of the vector bundle of leafwise half-densities 
|TjF|^/^ to vector bundles on G via the mappings s and r respectively, and |T^|^/^ = 
r*(|TJF|i/2) ^ s*(|TJF|V2)_ '^Yie line bundle \Tg\^/^ is the bundle of leafwise half-densities 
on G with respect to the natural foliation Q 0. 

The space C^{G, |T^|^/^) has the structure of involutive algebra (see, for instance, 
There is a natural ^-representation R of the involutive algebra G^{G, |T^|^/^) in L'^(M). 
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For any k G C^{G, \TQ\^^'^), the operator R(k) in L'^(M) is defined as follows. According to 
the short exact sequence the half-density vector bundle |TM|^/^ can be decomposed 

as 

For any 'j & G, 5(7) = x, r(7) = the corresponding linear holonomy map defines a map 

Given m G ^^(M) of the form u = m ® M2,mi G /.^(M, |rj^|i/2), ^2 e L'^{M,\HJ^\^/^), 
R{k)u G L'^{M) is defined by the formula 

R{k)u{x)= f A;(7)s*Mi(7) ®rf/i;_i[M2(s(7))], x G M. 



Proposition 2. For any k G C^°°(G, |T^|^/2) j ^ ^^"^(M)^ t/ie operator R{k) J f{t)e'^^dt 
is of trace class. Moreover, for any k G C^{G, ITQ]^^"^), the formula 

{e,, f) = tiR{k) j fit)e''''dt, f G 

defines a distribution 6k on the real line M, 6k G V{M.). 

Let jFjv be a foliation in N^J-", which is the horizontal foliation for the natural leafwise 
flat connection in N*J-' (the Bott connection). The leaf of the foliation J-'n through a point 
z/ G N*J^ is the set of all dh*{iy) G N*J^ such that 7 G G, r{-f) = where tt : N*T M 

is the natural projection. 

Denote by HJ^^^ the normal bundle to J-'n, HTn = T{N* J^) /TT^. For any z/ G N*T, 
the space TyN*T is a coisotropic subspace of the space T^T*M equipped with the canoni- 
cal symplectic structure, and T,yJ^N its skew-orthogonal complement, therefore, the normal 
bundle H^J^fq has a natural symplectic structure (see, for instance, 0). 

Given an operator A under the conditions (Al) and (A2) with the principal symbol a, let 

p be a smooth function on T*M homogeneous of degree one such that p(^) 7^ for ^ G T*M, 

which is equal to p = a^/^ in some conical neighborhood of N*J-', and ft the Hamiltonian 

~ 1/2 
flow of the function p. Define ap to be the restriction of p on N*T: ap = aj^ . The function 

Op coincides with the transverse principal symbol of any operator Pi G \&^(M, |TM|^/^) such 

that the principal symbols of and A are equal on N*J^. 

The holonomy invariance assumption on a a implies 

rfp(zy)(X) = 0, veN*T, XeT^J^N- (2.2) 



Using (|2.2|) and the fact that ft preserves the symplectic structure of T*M, one can easily 
check that the Hamiltonian flow ft can be restricted on N*J-'. The resulting flow will be 
denoted by ft- By definition, the flow ft depends only on the 1-jet of the principal symbol 
a on N*T, therefore, it doesn't depend on a choice of p and can be naturally called the 
transverse bicharacteristic flow of the operator A. 
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Since ft preserves the symplectic structure of T*M and TjF/v is the skew-adjoint comple- 
ment to TN*J-', ft maps leaves of the foliation J^n to leaves. In particular, the differential 
dft defines a map T^Tn ^/t{i^)-^Af a symplectic map H^Tj^ Hf^(^y)TN- 

We say that a point v G N*T is a relative fixed point of the diffeomorphism ft : 
N*J^ — > N*J^ (with respect to the foliation JF/v), if there exist 7 G G such that r{pf) = 7r(z/) 
and f-t dh*^{u) = u. 

For any t G M, denote by Zt the set of relative fixed points of ft- We also introduce 
the corresponding set in the cospherical bundle SN*T = {i^ G N*J^ : ap{y) = 1}: SZf = 
Zt^SN* . This set might be not closed, but, for any k G C^(G, |T^|^/^)), the corresponding 
part 

SZt.k = {ve SN*J^ : (37 G suppfc,r(7) = n{u))f^t dh;{u) = u} 

is closed. By the tranversal ellipticity of a a, the flow ft is transverse to J^n, therefore, the 
relative period set 7^. = {t G M : SZt,k 7^ 0} is a discrete subset of M. 

The following theorem was proved in |]IU|, but we will give an independent proof. 

Theorem 3. Given an operator A under the conditions (Al) and (A2) andk G C^{G, \TQ\^^'^), 
the distribution 6^ is smooth outside of the relative period set of the transverse bicharac- 
terictic flow ft . 

Let GjF^ denote the holonomy groupoid of the foliation jF^r. Gjr^ consists of all pairs 
(7, G Gjr X N*J^ such that r(7) = n^u) with the source map sn '■ Gj^^ — > N*J^, 3^(1, i^) = 
dh*^^), and the target map vjy : Gjr^ N*J-',riy('y,h') = v. There is a projection tiq '■ 
Gjr^ Gjr given by the formula T^cini ^) = 7- Put also GsN*r = Gjr^ fl r]^^{SN*J^). 

For any (7, u) G Gjr^, denote by dH(^^^^) the associated linear holonomy map: 

dH(^^^y) : Hdh*{u)^N — ^ HyJ^iq. 

It is easy to see that dH(^y u) preserves the symplectic structure of HJ^^. 

Denote by Q : TN*T HJ^^ the projection map. The differential of the map {tn, s^) : 
GjF^ N*J^ X iV*jF defines an isomorphism of the tangent space T(^^,^)Gjf^ with the set 
of all (fi,f2) G TyiV*jF © Tdh*{u)N*J^ such that the normal components of Vi and V2 are 
connected by the holonomy map: Q{vi) = dH{^^y){Q{v2)). 

The holonomy groupoid Gjc-^ has the natural foliation Qjr^ such that the tangent bundle 
T{'y^u)GrN corresponds to TyJ^^®Tdh*(^v)^N under the isomorphism described above. The nor- 
mal space H(^j^^)Qjr^ to Qj^^ is isomorphic to the set of all (f 1, ^2) G H^Tn © Hcih*(u)^N such 
that Vi = dH(^y ,y){v2), and therefore the maps dr^ (dsN) define isomorphisms of H(^y^^)Qjr^ 
with H^J^N {Hdh*{u)^N) respectively. 

Lemma 4. The set Zt is a saturated subset of N*J^, that is, it is a union of leaves of the 
foliation J-'n- 

Proof. By the holonomy invariance of p, the Hamiltonian vector field Sp with the Hamilto- 
nian p satisfies the identity 

dH^,,,^{Q{Ep{dh;{um = Q{Ep{u)), z/GiV*^, 7 e G, r{j) = n{u), 
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therefore, there exists a vector field Sp on Gj^^ such that 

rfsiv(Sp(7, u)) = Ep{dh*^{u)), drN{Ep{l, z/)) = Ep{u), (7, u) G G^^. (2.3) 
Let Ft be a fiow on Gj^^ generated by Hp. By (|2.3|) , we have 

/i o Tat = Tat o Ft, fto sn = sn o Ft, 
or, if we write Ft : Gjr^ ^ Gjr^ as Fj(7, i/) = (^^(7, i/), /j(z/)), 

/t(rf/.;(z/)) = rf/.^^(^,,)(/,(z/)). (2.4) 

Take any u E Zt with the corresponding 7 G G such r(7) = 7r(z/),/_f dhl^iy) = u. Let 
(71, i^) G Gjr^. Then we have 

Mdh;^{i^)) = dh*p^^^^^^){Mu)) by (H) 

= {dh*^,^^^^,^odh;odh;.,){dh;^{u)) 
= dh;,{dh;^{u)), 

where 7' = 7^"^ 070 Fj(7i, z/) that imphes dh*^^{u) G Zf. □ 
The relative fixed point sets Zt can be naturally lifted to the holonomy groupoid Gj^^^: 

Zt = {(7, y) G Gr^ : f.t dh;{u) = z/}, SZt = ZtH Gsn-t, 

By Lemma |, Zt = rN{Zt) = SN^Zt). 

Let us assume that Zt is a smooth submanifold of Gj^^. By Lemma H, the tangent space 
to Zt at a point (7, z/) G 2f contains a subspace F^^^^^Zt, which is the graph of the linear 
map dft{v) ■ TyJ^N Tah*{iy)^N = Tf^(^^)J^]^: 

F{y,u)Zt = {(^^1,1^2) G T^J^N X Td^^^)J^N ■■ V2 = dft{u){vi)}. 

Let 

H(^^y)Zt = T{^^^y)Zt/ Fl^^^y)Zt, H{^^^y)SZt = Tl^^^y)SZt/ F^^^y)Zt. 

Definition 5. Let t G M be a relative period of the fiow ft- We say that the fiow ft is clean 
on Zt, if: 

(1) Zt'm a. smooth submanifold of GjFjvi 

(2) the normal space H(^^^y-)Zt at any point (7, z/) G Zt coincides with the set of all (f 1, V2) G 
H{^^y)Qjr^ such that V2 = dft{u){vi). 
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Let |TjFjv|^/^ be the vector bundle of leafwise half-densities on N*T, and s*j^{\TTn\^^'^) 
and r^(|TjF;v|^/^) are the lifts of this vector bundle to vector bundles on Gjr^ via the map- 
pings sn and r^r respectively. Let |T^jf^|^/^ be the vector bundle of leafwise half-densities 
on G^j^: 

\TGrA"^ = r^(|T^^r/') ® s*j,{\TTn\"'). 

The projection tiq : Gjc-^ G defines a local diffeomorphism tt^ : Qj^j^ G, that induces a 
map 

Define a restriction map 

as follows. If p = fr%p, ® s^p2, / e C,°°(G^^), Pi, P2 e (M, \TTn\'/^), then 

i?2p(7,^) = /(7,^)pi(7,^)CM[P2(rf/i;(z^))], (7,^) e Zi, 

where the map df^{u) : |T/j(,^)jF;v|^/^ ^ iTyTNl^^"^ is induced by the linear map dft{u) : 
TuJ-'n — »■ Tf^(^^)J-'N- 

If the flow /i is clean, there is defined a natural density dp^ on H(^^^y)Zt, being the fixed 
point set of the symplectic linear map dHi^^ ^-^ odftiv) of the symplectic space H^Tn (see, for 
instance, 0, Lemma 4.3]). Dividing d^iz by dap, we get a density (ip52 on H(^^^^^SZt. 

Using the natural isomorphism 

\TSZt\ = \FZ\ ® \HSZt\. 

one can combine the densities Rzi^ck G C^{SZt, \FSZt\) and d^sz ^ C^{SZt, \HSZt\) to 
get a smooth density RziiQk dfisz on S'Zj. 

Let o"sub(^) denote the subprincipal symbol of A. Define cTsub(-P) = ^CL~^asuh{A) in some 
conic neighborhood of N*T. The restriction of crsub(-P) on N*J^ is equal to the restriction 
on N*!F of the subprincipal symbol of any operator Pi E \E'^(M, |TM|^/^) such that the 
complete symbols of Pi and A are equal mod 5*"°° in some neighborhood of N*J^. 

Theorem 6. Let t G M &e a relative period of the flow ft- Assume that the relative fixed 
point set Zt is clean. Then, for any k G C^{G, \TQ\^^'^) and for any r in some neighborhood 
oft, we have 

/+00 
a,{s,k)e'''^--'Us, (2.5) 
-00 

where: 

1. Zj are all connected components of the set SZt in Gsn*j^ of dimensions dj = dimZj-; 
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2. aj has an asymptotic expansion 

A;) ~ j i'""' 2_^aj^r{k)s~'' , s ^ +00 (2.6) 

r=0 

with ajfl given by the formula 

where aj denotes the Maslov index associated with the connected component Zj (see 
below for the definition). 

3 Reduction to the case when A is elliptic 

In this section, we will assume that A is an operator under the assumptions (Al) and (A2). 
We will use the classes \1/'"'~°°(M, JF, |TM|^/^) of transversal pseudodifferential operators 
(see for the definition) and the Sobolev spaces H'^{M) of half-densities on M. Put also 
'^*''^{M,J^, |TM|i/2) = U^^'"'^~(M,J^, |TM|i/2). 

By 0, the operator P = A^l"^ satisfies the following conditions: 

(HI) P has the form 
where: 

(a) Pi E \1'^(M, |TM|^/^) is a transversally elliptic operator with the positive, holonomy 
invariant transversal principal symbol such that the complete symbols of and A are equal 
mod S~°° in some neighborhood of N*J-'; 

(b) Ri is a bounded operator from L2(M) to iJ-^(M) andfor any G ^*-~(M,:F, \TM\^/'^) 
the operator KRi is a smoothing operator in L'^{M), that is, it defines a bounded operator 
from L2(M) to C°°(M, \TM\^/^). 

(H2) P is essentially self-adjoint in L'^{M) (with the initial domain C°°(M, ITM]^/"^)). 

Lemma 7 ([J10[|). Any operator P, satisfying the conditions (HI) and (H2), can be repre- 
sented in the form 

P = P2 + R2, (3.1) 

where: 

(a) P2 E '^^{M,\TM\^^'^) is an essentially self-adjoint, elliptic operator with the posi- 
tive principal symbol and the holonomy invariant transversal principal symbol such that the 
complete symbols of Pi and P2 are equal mod S~'^ in some neighborhood of N*J-'; 

(b) R2 is a bounded operator from L'^{M) to H-\M) and, for any K E ^'*'"°°(M, J^, \TM\^/'^), 
the operator KR2 is a smoothing operator in L'^{M). 
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Proof. Take a foliated coordinate chart f2 on M with coordinates (x, y) ^ P x f (/ is the 
open interval (0, 1)) such that the restriction of on f/ is given by the sets y = const. Let 
Pi G S^{I"' X R") be the complete symbol of the operator Pi in this chart. Assume that 
y, ^, 77) is invertible for any {x,y,^,ri) G U, + \ri\'^ > R^, where R > 0, U is a. conic 
neighborhood of the set 77 = 0. Take any function G C°^{I^ x R"),0 = (f){x,y,^,r]),x G 
P,y G J*^,^ G M.^,ri G M'', homogeneous of degree in {^,r]) for + |?7p > 1, which is 
supported in some conic neighborhood of 77 = and is equal to 1 in U, and put 

P2{x, y, e, v) = Mix, y, ^) + (1 - 0)(i + lei' + 1^1')'/'- 

Take P2 to be the operator P2{x, y, D^, Dy) with the complete symbol p2 (or, more precisely, 
P2{x,y, D^, Dy) + p2{x,y, Dx, Dy)* to provide self-adjointness) and put R2 = P ~ P2- The 
operator P1 — P2 has order —00 in some conic neighbourhood of N*J-', therefore, for any K G 
^*'-°°(M, J^, |TM|^/2) ^j^g operator K{Pi - P2) is a smoothing operator |^, that completes 
immediately the proof. □ 

Denote by W{t) = e**^^ the wave group generated by the elliptic operator P2. It is 
well-known that W{t) is a Fourier integral operator (see below for more details). Put also 
R{t) = (^'P - Wit). 

Proposition 8. For any K G ^*'"'^(M, J", \TM\^/'^), the family KR{t),t eR, is a smooth 
family of bounded operators from L'^(M) to C°°{M, ITM]^^"^). 

Proof. Since P^ = A & ^^(M, |TM|^/^), by interpolation and duality, P defines a bounded 
operator from H^{M) to L'^{M) and from L^{M) to H-^{M) and, for any natural N, P^ 
defines a bounded operator from H^{M) to L'^{M) and from L'^[M) to H~^{M). Since 
R2 = P-P2 and P2 G "^^{M, \TM\^^'^), R2 also defines a bounded operator from H^{M) to 
L^{M) and from L'^{M) to H~\M). 

By assumption, for any K G \E'*'~°°(M, JF, |TM|^/^), the operator KR2 is a smooth- 
ing operator, therefore, the operator KR2P^ is defined as an operator from H^{M) to 
C°°(M, |TM|i/2). 

Lemma 9. For any any G N, i/ie operator KR2P 

extends to a hounded operator from L?'{M) to C°°{M, |TM|^/^). 

Proof. We will prove the lemma by induction on N. For N = 0, the statement is true by as- 
sumption. Let us assume that it is true for some N, that is, for any K G '^*~°°{M, JF, |TM|^/^), 
the operator KR2P^ extends to a bounded operator from L'^{M) to C°°(M, \TM\^/'^). 

We have the equality - P^ = R2P + P2R2 as operators from H^{M) to H-^{M), 
therefore, for any K G ^*'-°°(M, J^, |TM|i/2)^ 

KR2P^+^ = K{R2P)P^ = K{P^ - P|)P^ - KP2R2P^. 

The operator P^ - P| g ^^(M, |TM|^/2) j^g^g ^^der —00 in some conic neighbourhood of 
N*J^, therefore, for any K G **-°°(M, JF, |TM|i/2) the operator K{P'^ - P|) extends to a 
bounded operator from H'^M) to C°°(M, iTMl^/^j g^^y ^ g^^^^ ^(^p2 _ p^^^pN extends 
to a bounded operator from L'^{M) to C°°(M, |TM|^/2)_ 
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Since P2 e ^^(M, \TM\^/'^) and K e ^* ~°°(M, J", \TM\^/'^), by the composition theorem 
1, fsTPs e ^*'-°°(M, JP", |TM|i/2) j^n^^ induction hypothesis, KP2R2P^ extends to a 
bounded operator from L'^{M) to C°°(M, |TM|i/2^. □ 

By the Duhamel formula, we have 

R{t)u = I [ e*"^2 R2 e'^'-^^^udr, u G H\M) C D{P), 
Jo 

therefore, for any K G ^*'"°°(M, J^, \TM\^/^), 

KRit) = i [ Ke'^^^ R2 e*(*~")-P dr = i [ e'^^^e-'^^^Ke'^^^ R2 e^^*"")^ dr. 
Jo Jo 

Any operator K G ^*'^°°(M, JF, |TM|^/^) is a Fourier integral operator (see below for more 
details) and, using the composition theorem for Fourier integral operators, one can check that 
e-irP2j^girP2 ^ ^*.-°°(M,JF, |TM|i/2)_ Therefore, the operator e-'^^^Ke'^^^R2 extends to 
a bounded operator from L'^{M) to C°°(M, |TM|i/2). Since e'^^^ maps C°°(M, |TM|i/2) 
C°°(M, ITMI"*^/^) and, by the spectral theorem, e**^*"'^-'^ is a bounded operator in L^(M), 
for any G \I'*'~°°(M, JF, |TM|^/^), the operator KR{t) extends to a bounded opera- 
tor from L'^(M) to C°°{M, |TM|^/^). Moreover, one can be easily seen from above argu- 
ments that the function KR{t) is continuous as a function on M with values in the space 
/:(L2(M),C~(M, |TM|1/2)) of bounded operators from L^{M) to C~(M, \TM\^/^). 
For any u G H^{M), the function R — if : t KR[t)u is different iable, and 

j^KR{t)u = iK{Pe''^u - P2e''^^u) = i{KP2R{t) + KR2^'^)u. 

The operator KP2R{t)+KR2e'^^ extends to a bounded operator from L'^{M) to C°°{M, \TM\^/^), 
and, moreover, the function t 1— KP2R{t) + i^i?2e**^ is a continuous function on M with 
values in C{L'^{M),C°°{M, |TM|^/^)). Using this, one can be easily seen that the function 
t KR{t) is differentiate as a function on M with values in C{L'^{M),C°°{M, ITM]^/"^)) 
and 

^KRit) = i(KP2R(t) + KR2e''^). 
dt 

Let us proceed by induction. Assume that, for any K G \E'*'~°°(M, JF, |TM|^/^) and for 
some natural n, the function KR{t) is n-times differentiable as a function on R with values 
in C{L'^{M),C°°{M, \TM\^/'^)) and the derivative KR^'^\t),t G M satisfies the equation 

{t) = ^irP2^^^"~^^ (t) + i''KR2P''-^e''^. (3.2) 

To prove that the function t ^ KR^^^ (t) is differentiable as a function on R with values in 
C{L^{M), C^{M, \TM\^/^)), as above, it suffices to prove that the derivative {d/dt)KR^''\t)u 
exists for any u from a dense subspace of L^{M), it extends to a bounded operator from 
L'^{M) to C°°(M, |TM|^/^), and its extension is continuous as a function on R with values 
in C{L\M), C^{M, \TM\^/^)). 
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From (p. 21), one can easy to see that the derivative {d/dt)KR^"\t)u exists for any u G 
H^{M) and satisfies the equation 

^KR^''\t)u = iKP2R^^\t)u + i^+^KR2P''e''^u. (3.3) 
at 

The first term in the right-hand side of (|3.3|) , iKP2R^^^ (t), is a bounded operator from L'^{M) 
to C°°(M, |TM|^/^) by the induction hypothesis. By Lemma ^ the operator KR2P" extends 
to a bounded operator from L'^(M) to C°°(M, |TM|^/^), and, by the spectral theorem, e**'^ is 
a bounded operator in L'^{M), therefore, the second term in the right-hand side of ( |3.3| ), the 
operator i'^+^KR2P'^e'^^ , extends to a bounded operator from L'^{M) to C°°{M, \TM\^/'^). 
It is also clear the right-hand side of (|3.3|) is continuous as a function on M with values 
in C{L'^ (M) , C°° {M ^ |TM|^/^)). This completes the proof of the existence of the derivative 
iri?("+^)(t) = {d/dt)KR^''\t) and the induction arguments. □ 

Proof of Proposition^. Let W{t) and R{t) be as in Proposition | and A; G C^{G, \Tg\^l^). 
Define 9k{t) by the formula 

Ok^t) = tr R{k)W{t) + tr R{k)R{t). 

Since P2 is an elliptic operator, the operator J f{t)e'^^^^dt is a smoothing operator in V{M) 
and the trace of the operator R{k)W{t) is well-defined as a distribution on M [§. Since any 
bounded operator T in L'^{M), which extends to a bounded operator from L?'{M) to H''{M) 
with s > n = dimM, is a trace class operator, the trace tr R{k)R{t) is a well-defined smooth 
function on M by Proposition ^ □ 

Corollary 10. For any k G C^°°(G, \Tg\^/^), the function ti R{k)R{t) = 9 k (t) -tr R{k)W{t) 
is a smooth function on M. 

It should be noted that, without any additional assumption about the operator P in 
question, the corresponding distribution on Gjr^ x M, tr R{k)R{t), might be very 

singular, but the singularities of the distribution k ti R{k)W{t) can be described rather 
explicitly under the clear intersection assumption. 



4 The case of an elliptic operator 

Let P2 G ^^(Af, |TM|^/^) be an essentially self-adjoint, elliptic operator with the positive 
principal symbol and the holonomy invariant transversal principal symbol and W{t) = e**^^. 
The singularities of the distribution t tr R{k)W{t) can be studied in a standard manner, 
using microlocal analysis. 

Fix k G C^{G,\Tg\^/^). We will consider the operator family R{k)W{t) as a single 
operator R{k)W from L'^{M) to L'^{R x M). We will prove that this operator is a Fourier 
integral operator. At first, let us recall well-known facts about the structure of the operators 
R{k) and W. 
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As above, let p be a smooth function on T*M homogeneous of degree one such that 
p(0 7^ for ^ G T*M, which is equal to p = a^^^ in some conic neighborhood of A^*JF and ft 
the Hamiltonian flow of p. Without loss of generality, we may assume that p is the principal 
symbol of the operator P2. Let Ap be the Lagrangian submanifold in T*R x T*M x T*M: 

Ap = {((t, r), (x, 0, {y, V)) e T*M X f*M xf*M:T = p{x, 0, {x, = f-t{y, v)}- 

Then W is a Fourier integral operator associated with the canonical relation A^, W G 
/-i/'*(M X M X M,A^). 

The operator R{k) belongs to ^°-°°(M,:r, \TM\^/'^) and, therefore, is a Fourier integral 
operator associated with an immersed canonical relation, which is the image of Gj^^ under 
the mapping 

(r^, Sn) : Gr^ T*M x T*M, (7, u) ^ (z/, dh^iu)), 

given by the source and the target mappings of the groupoid Gjr^ (see Q). More precisely, 
R{k) G I~P/\M X M,G'^^). 

By the tranversal ellipticity of p, the intersection of A^ with G'jr^ is transverse, and, by 
the composition theorem of Fourier integral operators [|lT], the operator R{k)W is a Fourier 
integral operator associated with an immersed canonical relation from T*M to T*(R x M) 
given by the map 

n : R X G^^ ^ T*R X T*M x T*M, (t, 7, u) ^ (t, j9(z/), i/, f.tdh;{iy)). 

More precisely, G /-p/2-i/4(k x M x M; R x G^j^^n). 

Recall that the trace functional can be treated from the point of microlocal analysis as 
follows H- Let A : R X M ^ R X M X M be the diagonal map, A{t, x) = (t, x, x), {t, x) G 
R X M, and tt : R x A/ ^ M the projection map. Then 

tr R{k)W = 'K,A*-Wk, (4.1) 

where Wk E C°°{R x M x M, |T(R x M x M)\^/'^) is the Schwartz kernel of the operator 
R{k)W, A* : C°°(R x M x M, |T(R x M x M)|i/2) ^ (:7°°(R, |TR|i/2) ® C°°(M, |TM|) is 
defined by the formula 

A*{si® S2® S3){t,x) = Si{t) {S2{x) ® Ssix)), t G R, X G M, 

where Si G G°°(R, \TR\^/^), S2 G C°^(M, |TM|i/2), ^3 e (:7°°(M, |TM|i/2)^ ^^^^ 
TT, : G°°(R, |TR|i/2) ^ c'°°(M, |TM|) ^ C°°(R, iTRj^/^) 

is given by integration along fibers of the projection it. 

It is known that tt^A* G /°(R x M x M x R, F), where F is the conormal bundle to the 
diagonal in R x M x M x R: 

F = {{t,Ti,Ui,U2,t,T2) G T*R X T*M x T*M x T*R : Ui = -U2,ri = -T2}. 
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There is a commutative diagram 



r ^ z 



P2 



(4.2) 



f 

T*(R X M X M) M X G:fn 

where 

Pi(t,7,z/) = (n(t,7,z/),t, -p(z/)) = {t,p{u),u,-u,t,-p{u)), (t,7,z/) G 

P2 is a natural inclusion and 

ip{t, r, z/, -I/, t, -r) = (t, r, u, -z/), (t, r, z/, -u, t, -r) G T. 

It is easy to see that ( |4.2| ) is a fiber product diagram, that is, 

Z = {ix,y) G (M X G^^) X r : n(x) = 

Using this fact and the functoriality properties of the wave-front sets (see, for instance, 
jl^), one get immediately the description of the singularities of the distribution 6k, given by 
Theorem |^. 

To finish the proof of Theorem |^, we will state under the assumption on the fiow ft to 
be clean in the sense of Definition ^ that TT^A*Wk is a Lagrangian distribution and compute 
its symbol. We begin with computation of the symbol of the operator R{k)W. Recall first 
the description of the principal symbol of the operator R{k). 

According to the short exact sequence 

^ Tg?^^ ^ TG^^ ^ Hg:p^ ^ 0, 
the half-density vector bundle on Gjr^ can be decomposed as 

where is the transverse half-density bundle on Gjr^: |^/^ = \H^Tn\^^'^ = 

\Hdh*{u)^N\^^'^ ■ 

Let \dy A dr]\^^'^ G C°^{N*T, \HJ^n\^^'^) be given by the Liouville form of the canonical 
transverse symplectic structure on the foliated manifold (A^*JF, jF/y), and r^(|(i?/ A dri\^/'^) G 
C°°(G^^, \Hg^^\^/^) its pull back via the map : G^^ ^ N*J^. 

Recall that the space S"^{Gjr^, |TGjf^ |^/^) is defined to be the space of all smooth sections 
s of the vector bundle \TGjr^\^^'^ on Gj^^ homogeneous of degree m such that 7rG(supp s) is 
compact in Gjf. 

The half-density principal symbol of R{k) is an element of S'^{Gyr^, iTGjr^l^^"^) given by 
the formula 



a{R{k)){j, u) = n*ak{j, u) ® r],{\dy A rfr/l^/^), (7, y) e G 
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The Maslov bundle M(M x Gjr^,n) of the immersed canonical relation (M x Gjr^,n) 
restricted to t = is isomorphic to the Maslov bundle M(Gjc-^) of Gjc^, therefore, it has 
a canonical constant section, which extends to a global section s of M(]R x Gjr^,!!) by 
requiring it to be constant along each bicharacteristic (t, r, z/i, z/2), z/i = f^t{j^2),t G M. 

Using the description of the principal symbol of the operator W{t) given, for instance, 
in and the composition theorem of Fourier integral operators, we get immediately that 
the principal symbol of the operator R{k)W is an element of ^^(IR x Gjf^, M{R x Gjt^, 11) 
|T(R X GjF^)|^/^), whose value at a point (t,7, z/) G M x Gjf^ is given by 

a{R{k)W){t, 7, z/) = e^/o --b(P)(/-.d?«;H)d.g ^ |^^|i/2 ^ ^ ^^^|^^ rfr^r/^). 

Now let us turn to the composition ( [4.1|) . First, we check the corresponding cleanness 
assumption. 

Lemma 11. The assumption on the flow ft to be clean on Zt in the sense of Definition ^ 
guarantees that the composition o/M x Gyr^ with F is clean. 

Proof. By definition, the composition of M x Gj^^ with F is clean iff Zt is a submanifold of 
M X Gjr^ and in addition the fiber product diagram ([4.2|) is clean at any point {t, 7, u) G Z, 
that is, the linearized diagram 

dpi 



Tp-^{t,'y,u)'^ 



dip 



dp2 



(4.3) 



T(t,p(.),.,_.)(T*(Mx Mx M)) 



dn 



is a fiber product diagram. Since T(^t,-y,u)2^ is always contained in TyN*J^ © Tdh*{u)N*J^, this 
is true iff the diagram 



dip 

T^t,pM,u,-.)iT*Rx N*J^ X N*J^) 

is a fiber product diagram. 

The diagram ( [4.4|) has a subdiagram 



dpi 



dp2 



(4.4) 



dn 



-^pi (t,^,u) r 



dpi 



L 



,z 



dn 



where L 



(i,7,i/)l 



d(/3 

X G^^) c T^t,f,u){^ X G^^) is given by 



dp2 



(4.5) 



N 



X G^^) 



-^(t,7,y)(I^ X Gjr^) 

= {(t/,Vi,r2,H^)GTi( 



TV 



Tdh*{u)^N 
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Z C T(t^-f^y)Z by 

L^tn,u)Z = {{U, Vu V2, W) e Tt{R) © T.J^n © Tdh*^{u)J^N © H^-y,u)Q^M 

,U = 0,V, = -df.tidh;{u)){V2), w = o} = nj^M 

and Lp^(^t,j,u)^ C Tp^(^t,y,u)^ by 

LMt,^,u)r = {{Ui, Vi, V2, U2) e T(t,p(,))(T*M) © T.J^N © T^uJ^n © T(t,_p(,))(T*M) 

:f/i = f/2 = 0,\/i = -\/2} 

which can be easily seen to be a fiber diagram. 

Therefore, the diagram ( [4.4|) is a fiber product diagram iff the quotient diagram is a fiber 
product diagram: 



dip 



dp2 



(4.6) 



where 



= {(f/, 1^1, \/2, VT) G Tt(M) © T.J-^ © T,^;(,)^;v © ^(7,.)6^.F^. : 1^1 = 0, 1^2 = 0} 



N, 



HMt,y,u)T = {{U^, V^, V2, U2) e T(t,p(,))(T*M) © H,Tn © H_,T^ © T(i,_p(,))(r*M) 

■.Ui = -U2,Vi = -V2}. 



In its turn, the diagram ( [4.6|) is a fiber product diagram iff the flow ft is clean on in the 
sense of Deflnition ^. □ 

For any connected component Zj of Zt, the excess of the clean diagram ([4.2| ) equals c/j, 
the dimension of the relative flxed point set SZj in GsN*r- By the composition theorem of 

Fourier integral operators, 9k belongs to 0^/^ ^(^t); where = {(t,T) G T*M. : r G 
]R_}, that proves the desired representation of 6^ in the form ( p.5|) and the existence of the 
asymptotic expansion ( pl6| ) for aj(s, /;;).[] 

To obtain the explicit formula for the leading coefficients aj^, we compute the principal 
symbol of 9k, cr{9k), following the arguments in [0. 

^Note that the appearance of the term — p/2 in the exponent is due to the fact R{k)W G /^^/^^^/''(M x 
Af;Rx G^„,n). 
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Fix a connected component Zj of Zt and (t, 7, v) G Z,j. The fiber product diagram 
defines a composition map |ll 



Tp,(t,^,.)r|i/2 ® |T(t,^,,)(M X G^^)!^/' ^ |T(t,_p(,))(AOr/' ® \T{ta,v)Z,\ (4.7) 



and, due to ( |4.1|) and the composition theorem for Fourier integral operators, the value of 
the principal symbol oiQ^^^ E |TAt|^/^ at a point (t, t) G is given by integration over Zj 
of a(7r,A*) * alWk) G C"^(At x Z^-, ITA^I^/^ ® \TZj\). 

Using the functoriality of the * operation on half densities with respect to reduction (cf. 
p!3| , proof of Lemma 4.6]), the computation of the ^-product a{n^:A*)*a(Wk) can be reduced 
to the computation of a *-product defined by the transverse fiber product diagram ( |4.6|) : 

*t : |i/p,(t,^,,)r|i/2 ^ \H^^^^^^)(m X G^^)|^/2 ^ \T(^t,-piu))At\'^^ ® l^(t,7.-)^j-|- 

More precisely, we apply the result stated in [^, proof of Lemma 4.6] with a symplectic 
vector space V = T(^t,p(u),u-u,t~piu))iT*R x T*M x T*M x T*R), two Lagrangian subspaces 
in V: Ai = Tpj(t,^,i/)r and A2, which is the image of T(^t,'y,u)(M ^ ^:Fn) ^ reduction 
given by the coisotropic subspace F = T(^t,p{u),i'-u,t-piu)){T*R x N*J^ x N*J^ x T*R). 

By this result, the leafwise component of cr{9k), criiOk) G \L(^t,'y,v)Zj\, is obtained from the 
leaf wise component of a{R{k)W): 

by application of the restriction map Rz'- 

and the transversal component of a{6k), <Jt{Ok) G \T(^t,-p(u))i^t)\^^'^ ® \H{t,'y,i^)Zj\, is equal to 
the *rPi'oduct of the transverse component of (T{R{k)W), 

atiRik)W) = \dt\^/^ ® r*j^{\dy A dr^l^/'), 

and the transverse component of cr(7r^.A*). By we get 

that completes the calculation of the half-density principal symbol of 6k and implies the 
formula ( p.7|) for the leading coefficients aj^ as in 



5 Maslov indices 

In this section, we define the Maslov factors a, corresponding to (7,2^) G Zt. For this 
goal, we will use local coordinates on the holonomy groupoid G described, for instance, in 
P, |[. Choose a pair of compatible foliated charts near the points 7r(z/) and 7r(/((z/)) with 
the coordinates {x,y) and {x',y), corresponding to 7 G G. Then we have the corresponding 
coordinates in HyT^ = TyN* T /T^J^m defined as {6y,6ri), and the vertical and horizontal 
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subspaces V„ and H„, given by the equations 6y = and drj = accordingly. The hnear 
holonomy map dH(^^y^ of the fohation {N*J^, Tn) defines an isomorphism of the symplectic 
spaces Hf^(^^')J^N and Hi^J^n, which preserves the vertical and horizontal subspaces. Due to 
this isomorphism, we can obtain a closed curve 1^(7,1/) in the Lagrangian Grassmannian Q of 
the symplectic space H^J^^, pulling back, via dft, the vertical subspace at /t(z^) for t between 
and T. Denote by K{'y,u) the intersection number of u with the horizontal subspace H^: 



K 



Let x(i:, x, I/, ^, 77) be the generating function of the canonical transformation ft in the 
chosen coordinates. Recall that x is the solution of the Cauchy problem 



dtX = P{x, y, 4x, dyx), x(0, x, y, ^, r]) = + yr]. 



(5.1: 



By the holonomy invariance of p, it can be easily seen that x{tjX,y,0,ri) is independent of 
X and x, y, 0, 7]) = y, v)- Let 



dyyX dyj^X 1 

d-nyX d^^x 



-1 







We define the Maslov factor (7(7, z/) as 

a(7, v) = sgn R^.^^^) + 2/t(^,^), (7, u) e Z. 

It is clear that (7(7, u) is a locally constant function on Z. 

To handle with Maslov factors in the proof of Theorem |^, let us write the Schwartz 
kernel of the operator R{k) in a foliated coordinate chart on G given by a pair of compatible 
coordinate systems as k{x,Xi,y)6{y — yi), and the Schwartz kernel W{t) microlocally as an 
oscillatory integral of the form 



a{t,xi,yi,X2,y2,^,v)d^dr], 



where 



xi, yi, X2, 1/2, ^, v) = X{t, xi, yi, ^, rj) - - ^2^? 

and x{tjXi,yi,^,ri) is the generating function of the canonical transformation ft given by 
Then the Schwartz kernel of the operator R{k)W is given by the formula 

^jait,.uyux2,y2,i,v)^^^ xi, yi)a{t, Xi, yi, X2, Z/s, ^, v)dxid^d7], 



from where one can easily derive the desired assertion, following the arguments of 
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